1 research outputs found
DEPLOYR: A technical framework for deploying custom real-time machine learning models into the electronic medical record
Machine learning (ML) applications in healthcare are extensively researched,
but successful translations to the bedside are scant. Healthcare institutions
are establishing frameworks to govern and promote the implementation of
accurate, actionable and reliable models that integrate with clinical workflow.
Such governance frameworks require an accompanying technical framework to
deploy models in a resource efficient manner. Here we present DEPLOYR, a
technical framework for enabling real-time deployment and monitoring of
researcher created clinical ML models into a widely used electronic medical
record (EMR) system. We discuss core functionality and design decisions,
including mechanisms to trigger inference based on actions within EMR software,
modules that collect real-time data to make inferences, mechanisms that
close-the-loop by displaying inferences back to end-users within their
workflow, monitoring modules that track performance of deployed models over
time, silent deployment capabilities, and mechanisms to prospectively evaluate
a deployed model's impact. We demonstrate the use of DEPLOYR by silently
deploying and prospectively evaluating twelve ML models triggered by clinician
button-clicks in Stanford Health Care's production instance of Epic. Our study
highlights the need and feasibility for such silent deployment, because
prospectively measured performance varies from retrospective estimates. By
describing DEPLOYR, we aim to inform ML deployment best practices and help
bridge the model implementation gap